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Abstract: In this paper, we report a method for 3D visualization of a 
biological specimen utilizing a structured light wide-field microscopic 
imaging system. This method improves on existing structured light imaging 
modalities by reassigning fluorescence photons generated from off-focal 
plane excitation, improving in-focus signal strength. Utilizing a maximum 
likelihood approach, we identify the most likely fluorophore distribution in 
3D that will produce the observed image stacks under structured and 
uniform illumination using an iterative maximization algorithm. Our results 
show the optical sectioning capability of tissue specimens while mostly 
preserving image stack photon count, which is usually not achievable with 
other existing structured light imaging methods. 

© 201 1 Optical Society of America 

OCIS codes: (180.6900) Three-dimensional microscopy; (180.2520) Fluorescence microscopy; 
(1 10.6880) Three-dimensional image acquisition; (100.3020) Image reconstruction-restoration 
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1. Introduction 



Wide-field optical microscopy is a robust and efficient imaging method but has no depth 
sectioning capability. The 3D optical transfer function of wide field imaging shows that high 
spatial frequency structures are always in-focus while low frequency (especially zero 
frequency) components are always transmitted from every depth (the "missing-cone" 
problem) [1]. In contrast, scanning microscopy methods, e.g., confocal microscopy and two- 
photon microscopy, has 3D capability but their frame rates are often limited by the pixel 
residence time required to generate images with sufficient signal-to-noise ratio (SNR). 
Powerful methods for 3D resolved imaging based on structured light illumination have been 
developed recently by several groups allowing wide-field visualization of the focal plane 
while rejecting out-of-focus background "haze" [2-6]. When a single spatial frequency grid 
pattern is projected onto the sample, the microscope efficiently image only those part of the 
grid patterns that lies within its focal depth. Various demodulation approaches have been used 
for the reconstruction of optical sectioned image. There have been frequent attempts to 
achieve higher resolution in both transverse and axial dimensions using structured 
illumination in optical microscopy (bright-field and fluorescence). Based on the recorded 
images corresponding to the different phases of the projected grid, Wilson and associates have 
demonstrated 3D optical sectioning [3] and Gustafsson and associates have shown spatial 
resolution enhancement [4]. Essentially, these methods require recording of minimum three 
structured images corresponding to precise phase shifts of the grid. Most recently, Mertz and 
associates presented a new approach, called the HiLo microscopy [5], that combines the in- 
focus high frequency content extracted from a uniformly illuminated image and the in-focus 
low frequency content extracted from a structured light illuminated image to generate an 
optically sectioned image. While these methods are able to improve image contrast, the loss of 
out-of-focal plane photons limits the final image signal-to-noise ratio. Additionally, if the 
contrast of the projected grid is low this reduces the modulation and provides the weaker 
sectioning capability and as a result reduces the intensity of the optically sectioned image [6]. 
Therefore, the applicability of these methods is limited for the imaging thick, turbid biological 
tissues where the number of photons is already limited because of scattering. In this paper, we 
proposed a method, extending structured light 3D imaging methods, such as HiLo 
microscopy, which can assign photons generated from off-focal plane excitation improving 
final image SNR. Our method seeks to better utilize these "lost" photons by using the 'prior 
knowledge' about the optical transfer function of the structured light illumination. While this 
method may be applied for a variety of contrast mechanisms, we focused here on fluorescence 
based imaging. Utilizing a maximum likelihood approach, we identify the most likely 
fluorophores distribution in 3D that will produce the observed image stacks under structured 
and uniform illuminations using an iterative maximization algorithm. We further demonstrate 
that this approach provides comparable background rejection as existing structured light 
imaging methods, such as HiLo microscopy, while improving final image SNR. 

2. Mathematical formulation of algorithm 

Figure 1 shows the schematic of structured light imaging based wide field fluorescence 
microscope system. Consider the coherent light, wavelength \ and wave number k x , which 
is transmitted by a one dimensional sinusoidal grating (spatial frequency k g ) is used for 
fluorescence excitation under structured illumination. Let A(x, y, z) be the fluorophores 

distribution function in the specimen. For specimen positioned at axial displacement location 
z , the emission intensities corresponding to uniform and structured illuminations are defined 

as Ay(x, y,z,z) = Mx,y,z,z)xU(x,y,z) and X s (x, y, z, z) = /L(x, y,z,z) x S(x, y, z) , 

respectively, where U(x, y,z)= I E u (x, y, z) I 2 and S(x, y, z) = \E s (x, y,z)\ 2 are the 
projected uniform and structured light intensity distributions at the object space respectively. 
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The 3D distribution of electric fields, assuming sinusoid grid focused at object plane (£,77) , 

corresponding to uniform and structured excitations can be calculated, using the Fresnel 
equation [7], as 



e 

E u (x, y, z) = — ffexp 
i\z JJ 



{x-£f +{y-rjf) 



d^drj 
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Fig. 1 . Schematic of structured light imaging based wide field fluorescence microscope system 

For uniform illumination, the intensity function of the detected fluorescence photons at the 
CCD plane can be written as [8] 



f u (x\y\z) = {^ j A v (x,y,z,z)\h iD (x'-x,y'-y,z'-z)\ 2 )\ z=0 



(2) 



Here h iD is the 3D point spread function (PSF) of the system. Similarly, for structured 
illumination the intensity function of the detected fluorescence photons can be written as 



f s (x\y\z) = X s {x,y,z,z)\h 3D (x'- x,y'- y,z'~ z)\ 2 ) L 0 



(3) 



Uniform and structured excitation patterns are focused at multiple z-planes, by moving the 
object in axial direction, to generate the full 3D image data. 

Let N v (x ', y ', z ) be the number of photon detected at the pixel coordinate (x\y : ) with 
uniform illumination focused at axial displacement z of the object. Since it follows the 
Poisson distribution, the log-likelihood function can be written as [9] 



l u (A(x,y,z)) = X [N u {x\y\z)Mf u {x\y\z)}- f u {x\y\z)} 



(4) 



Similarly, let N s {x\ y\ z) be the number of photon detected at pixel (x\ y') with 

structural excitation pattern focusing at depth z ■ Since it also follows the Poisson 
distribution, the log-likelihood function is 



l s (Mx, y, z)) = £ [N s (x\ y ', z) ln{/ s (x\ y ', z)} - f s (x\ y \ z )] 



(5) 



The combined log-likelihood function for uniform and structured illuminations can be 
written, by using Eq. (4) and Eq. (5), as 
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l(A(x,y,z))=J][N u (x\y \ z ) In { f v (x \y\z)} + N s (x ', y ', z) In { f s (x ', y\z)} 

(6) 

-f u (x\y\z)-f s (x\y\z)] 

Images represented by f v (x , y , z) do not provide depth sectioning because of the presence 
of out-of-focus blur and thus not suitable for 3D imaging applications. As discussed before, 
optical sectioned image can be generated by using demodulation method that requires the 
uniform f u (x\y\z) and structured f s (x\ y',z) images [5], but the loss of out-of-focal plane 
fluorescent photons limits SNR. 

Our methodology provides maximization of log-likelihood function to identify the most 
likely X(x, y,z) by utilizing an iterative maximization algorithm. The maximization process 
provides the reassignment of the photons originally distributed in the wide field uniform 
image and lost in process of optical section image generation. Since the fluorophores 
distribution in 3D can already be known from using existing structured light reconstruction 
algorithms, this known distribution is used as the 'first guess' for the maximization algorithm 
that iteratively improves the SNR of this distribution. In such case, maximization can be 
performed by any fast numerical optimization method. We used Newton's method used for 
minimizing least square criteria or for maximizing likelihood function and the iteration step 
can be written as [10], 

\dl{2,(x,y,z)} I 3/t(x,y,z)l > ?<*>, 
[d 2 l{Mx,y,z)}/dA 2 (x,y,z)] 

The solutions of the first and second order derivatives of the log-likelihood function, 
defined in Eq. (7), are given as 

Dl(Mx,y,Z)) V rrr/ -Ml -m2 

- }][U(x,y,z)\h 3D (0,0,-z)\ 2 



dMx, y, z) 
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r}] 



{( X X s (x,y,z,z)\h, D (x'-x,y'-y,z'-z)\ 2 )U} 2 

x\y',z' 

The maximization process updates the new estimate X (k+[) (x, y,z) of the object function 

for iterative step k. The accuracy of the reconstruction partly depends on the smoothness of 
initial estimate chosen and the constrained parameters, i.e., positivity of the fluorophores, for 
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convergence of the algorithm. Total log-likelihood function is calculated for each iteration 
process and it is used for the convergence behavior. The values of log-likelihood function 
increase with iteration number and then its convergence decides the numbers of iteration 
needed for maximization. Figure 2 provides the summary of the proposed photon 
reassignment algorithm. 
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Fig. 2. Algorithm of photon reassignment process. 

3. Results and analysis 

3.1. Imaging of fluorescent beads 

The performance of the proposed method was evaluated with a z-stack of wide-field 
fluorescence images. The sample was prepared using 6/um size fluorescent, with an 

excitation wavelength of 480nm and an emission wavelength of 530nm . We prepared 
180/zot thick agarose gel (2%) specimen with embedded beads. A stock solution of beads, 

with initial concentration of 2. 1x10 s particles/ml, was diluted into DI water and the 
concentration of beads was adjusted to 200 : 1 that produced sufficient bead density for 
imaging multiple beads in the observed volume. 

Fluorescence excitation was performed using a diode laser of wavelength 
488nm (Sapphire 488HP, Coherent Inc., Santa Clara, CA, USA). For structured illumination, 
a Ronchi ruling was used. After passing through the excitation optics, the resultant spatial 
frequency of the grid at the object plane was 320 line pairs I mm . The imaging is performed 
using a water immersion microscope objective, 20x, NA 1.0 (Zesis). Long pass filter (BLP01- 
488R-25 from Semrock, USA) used for fluorescence signals imaging. Effective pixel size at 
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the sample plane was0.202//m . Axial positioning of the object was controlled by using the 
piezo-actuated stage (MAX314-3-Axis nano Max Stage, Thor Labs) and uniform and 
structured images are recorded corresponding to the 90 /urn depth range with 3/vm step size 

using the CCD camera (MEGAPLUS, Precision Instruments, Trenton, NJ, USA) and stored 
on a computer. Ten sets of uniform and structured images were recorded at each axial position 
throughout the 90 jum depth range. 



• • 


# # 


* 

(c) 


• 

(a) 


• 

(b) 


(d) 


(e) 


(0 


(g) 


00 


1 

(0 


• • 


• # 


• 

(0 


(i) 


• 

(k) 


• • 

t 

(m) 


• # 

(n) 


(o) 



Fig. 3. (a)-(c) Uniform and (d)-(f) structured images of fluorescence beads at object axial 
positions 24 jim, 48 jim, and 79 jim, respectively; (g)-(i) are the corresponding HiLo 
microscopy images, (j)-(l) are the reconstructed images using proposed model, and (m)-(o) are 
the deconvolution images. 

The uniform and structured image for three axial positions of the specimen at lA/um , 
A%/um and 19/jm are shown in Figs. 3(a)-(c) and 3(d)-(f), respectively. The HiLo images 
are shown in Figs. 3(g)— (i). We considered the 1 50 x 150 x 150 sample points in the volume 
which provide object size approximately 40 jum in x and y dimensions and 90 jum in z 
dimension. To perform the maximization process, we first numerically simulate the excitation 
intensities corresponding to uniform and structured patterns using MATLAB™. These 
excitation patterns were simulated with same number of sample points corresponding to each 
plane, i.e., there were a total of 30 3D excitation matrices (the size of each matrix is 
150x150x150) for structured excitation patterns and same for uniform. 3D PSF for the 
microscope objective was also simulated [11]. First, we calculated the HiLo images 
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corresponding to each axial plane and then used these images as the first prior for the object 

function of the corresponding plane X m (x, y, z) . The maximization is performed, as 

described in Fig. 2, and the reconstructed images are shown in Figs. 3(j)-(l) for the three 
aforementioned planes. Figures 3(m)-(o) are the deconvolution images of the same axial 
planes. Custom maximum likelihood deconvolution is carried out using information from 
uniform illumination images with positivity constraint. The obtained results are comparable 
with commercial Imaris deconvolution package (Bitplane Inc., Zurich Switzerland) although 
commercial package appears to include certain smoothness constraint. To observe the 
maximization process for our approach, values of log-likelihood function was calculated as a 
function of iteration cycle and we observe the convergence starts after around 30 iterations. In 
contrast, the deconvolution method took around 200 iterations (non-blind deconvolution). 
From the reconstructed results, it is apparent that the proposed method provides improvement 
in image quality by effectively increasing the in-focus signal while removing the out-of-focus 
background. 

The performance of reconstruction results are quantitatively compared with the images 
acquired using HiLo method by plotting the line profile, as shown dotted line in Fig. 3(c). The 
plot of intensity values is shown in Fig. 4(a). It can be clearly observed that the HiLo method 
effectively rejects the background photons of the uniform image improving image contrast. 
However, the proposed approach increases the number of the in-focus photons by assigning 
photons generated from out-of-plane excitation to appropriate locations and simultaneously 
rejects the background photons and hence improves the SNR and signal-to-background ratio 
(SBR) of the resultant image. 

Analysis of SNR and SBR was performed for the proposed method. We recorded 10 sets 
of uniform and structured images at each axial position for 90um depth range with 3um step 
size. Maximization was performed independently for each set using the proposed approach. 
Plot of in-focus signal (in-focus beads region) and background for an axial plane is shown in 
Fig. 4(b). It can be clearly observed that the maximization provides significant improvement 
for the in-focus signal while keeping the background level similar as for HiLo image. The 
improvement in the SBR for the reconstruction can be clearly observed and it is shown in Fig. 
4(c). 

The image SNR of the fluorescence beads is calculated as 



SNR=- 



(10) 



where / and tr, represent the mean and standard deviation of the in-focus intensities of the 
bead image. It is important to note that recorded beads image under uniform illumination 
includes background noise and this background need to subtract for calculation of SNR. 
However, both HiLo and maximized images are subjected to background suppression. Table 1 
shows the comparison of SNR, SBR and image contrast for an axial plane. 
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Fig. 4. (a) Line intensity plot showing the performance of the proposed method, (b) Analysis of 
the in-focus and background signals, (c) SNR and SBR plots. 
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Table 1. Evaluation of image parameters 





HiLo 


Maximized 


Uniform 


SNR 


53.37 


62.07 


54.88 


SBR 


11.8 + 0.61 


16.1+2.33 


3.3+0.05 


Contrast 


0.91 


0.93 


0.43 



Several conclusions may be drawn from this table. First, the Maximized method improves 
SNR and SBR metrics compared to the HiLo method, while contrast remains mainly 
unaffected; this is because in the Maximized method the signal and noise statistics are taken 
into account explicitly, and therefore the available photons are utilized more effectively. 
Compared to uniform illumination (bright field), the contrast improvement is similar for HiLo 
and Maximized, while the SBR improvement is even more drastic for the Maximized case. 
However, in terms of SNR HiLo underperform compared to uniform illumination; this could 
be because of the filters applied to HiLo effectively removes photons from the final image 
and, hence, the final image is subject to noise fluctuation. For the Maximized method, the 
recovery of in-focus photons increases the maximized image SNR and is superior to both 
uniform illumination and HiLo method. The analysis at other axial planes and a comparison of 
the results with deconvolution method is provided in the supplementary material. 

3.2. Imaging of thick tissue from zebra fish 

We have used fluorescently stained zebrafish intestine tissue sample for imaging in our wide- 
field fluorescence microscope. Adult zebrafish was dissected to extract the intestine sample. 
The sample was cleaned, fixed with 4% paraformaldehyde (PFA) and permeabilized with 
0.2% tritonX-100. Tissue sample was stained with Phalloidin with Alexa-488 as a fluorescent 
marker for labeling actin. For structured illumination, the spatial frequency of the grid at the 
object plane was 320 line pairs I mm . The imaging was performed using water immersion 
microscope objective, 20X, NA 1.0 (Zesis). Images were captured with a CCD camera and 
stored on a computer. Uniform and structured image for 30 axial planes were recorded with 
depth steps of 3 jum . 
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Fig. 5. (a) and (d) Wide field fluorescence uniform images of zebrafish intestine; (b) and (e) are 
the corresponding reconstructed images, for tissue axial positions 36 /Jtn and 51 pan, 
respectively; (c) and (f) show the line profile of intensity values 

Figures 5(a) and 5(d) illustrates the wide field uniform fluorescence images taken for the 
axial positions of the specimen at 36/um and 51/um , respectively. As discussed before, it is 
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important to note, due to high scattering effect, the contrast of projected grid reduces with an 
increase in sample depth. This reduces the modulation and provides the weaker sectioning 
capability for HiLo processing and result as reduce in number of photons used to reconstruct 
the HiLo image. The reassignment of the photons using the proposed approach is thus 
particularly significant in such situation. Figures 5(b) and 5(e) show the reconstructed images 
at 36jum and 51/um respectively. In order to determine the efficiency of proposed approach, 

the results are compared with the HiLo images by plotting the intensity line profile (red dotted 
lines in Figs. 5(a) and (c)), which is shown in Figs. 5(c) and 5(f), respectively. The 
improvement in SNR of the reconstructed images compared to the HiLo images is quite 
apparent. By assigning out-of-plane signal to the correct locations, the in-focus image is 
strengthened while the background is reduced. The utilization of the 'prior knowledge' about 
optical sectioning associated with the structured illumination provides the accuracy of photon 
reassignment for the correct 3D spatial location and prevents noise amplification during 
iteration process. 

4. Conclusions and outlook 

We reported a method for 3D visualization of the structured light wide-field fluorescence 
microscopic imaging system, which provides better SNR and SBR by reassigning 
fluorescence photons generated from off-focal plane excitation. Novel methods for 3D 
resolved imaging based on structured light illumination have been developed by several 
groups [2-6] allowing wide-field visualization of the focal plane while rejecting out-of-focus 
background "haze". While these methods improve image contrast, the loss of out-of-focal 
plane fluorescent photons limits image SNR. Our proposed method seeks to better utilize 
these "lost" photons by using the 'prior knowledge' about the optical transfer function of the 
structured light illumination. Since we observed improvement in both SNR and SBR we 
speculate that the proposed method should be useful to image deeper into biological tissues. It 
should be noted that this approach is different from typical deconvolution methods. The 
fluorophores distribution in 3D is already known from using existing structured light 
reconstruction algorithms although with limited SNR. This known distribution is used as the 
prior for the maximization algorithm that iteratively improves the SNR of this distribution. 
We demonstrate that the proposed method numerically maximizes with an order of magnitude 
fewer iterations than standard deconvolution [12]. More importantly, the proposed method 
should converge to the actual 3D structure with better fidelity since axial information is not 
lost in the "missing-cone" and will be evaluated in future studies. This new technique 
effectively accounts for out-of-focus excitation due to the lack of depth sectioning capability 
of wide-field microscopes. However, this technique does not take into account of out-of-focus 
signal due to tissue scattering of the excitation and the emission photons that may be 
incorporated in future algorithms. Other promising future research directions include the 
incorporation of this photon reassignment method into other wide field tissue imaging 
techniques such as various light-sheet based microscopes. For the presented results the 
processing time was around 1 hour for maximization of entire volume (150x150x150 pixels) 
by using the Intel(R) Core(TM) i5 CPU 3.33GHz processer with 4GB RAM, while the known 
3D excitation patterns matrix and 3D PSF were calculated prior to maximization. Our future 
works will be focused on improvement of computational speed by utilizing the parallelized 
computing. 
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